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1 . We the undersigned, Patrick Chiu, Donald G. Kimber, John Steven Boreczky and Andreas 
Girgensohn, declare s follows: 

2 We are employees of Fuji Xerox Palo Alto Laboratories and specialize in computer 
software development. 

3. We are the inventors of the invention described and claimed in the above U.S. patent 
application. 

4. For convenience, embodiments of this invention hereafter are referred to as the "Lite 
Minutes system". 

5. For convenience, Claim 1 of the above application is set out below with elements (a)-(f) 
identified as follows: 

A method for creating multimedia meeting minutes, comprising the steps of: 

(a) receiving a notation from a notetaking user during a meeting; 

(b) automatically recording an index value for the notation, the index value based on the 
context of the notation; 

(c) receiving a quantity of multimedia information from at least one multimedia source; 

(d) automatically selecting at least one portion of the quantity of the multimedia information 
based on the index value of the notation; 

(e) automatically creating an association between the notation and the selected portion of the 
quantity of multimedia information, where the association enables access to the selected 
portion of the quantity of multimedia information; and 
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(f) storing the notation and the association for retrieval at a future time, where the future time 
is a time during the meeting, wherein a single action by the notetaking user initiates the 
steps of receiving the notation, recording, selecting, creating, and storing. 

6. The "Lite Minutes system" prototype consists of "LiteMinutes.java" and "DataStore.exe" 
and software modules as listed in Exhibits 6 A'-'C\ 

7. The "Lite Minutes system" prototype embodiment of the invention contained all the 
elements of Claim 1 as described and claimed in the above U.S. patent application 

8. The "LiteMinutes system" is able to receive a notation from a notetaker during a meeting 
as claimed in Claim 1, element (a). 

9. The "LiteMinutes system" is able to automatically record an index value for the notation as 
claimed in Claim 1, element (b). 

10. The "LiteMinutes system" is able to receive multimedia information from a multimedia 
source as claimed in Claim 1, element (c). 

11. The "LiteMinutes system" is able to automatically select a portion of multimedia 
information as claimed in Claim 1, element (d). 

12. The "LiteMinutes system" is able to automatically create an association between a notation 
and selected multimedia information as claimed in Claim 1, element (e). 

13. The "LiteMinutes system" is able to store a notation and the association for retrieval at a 
future time as claimed in Claim 1, element (f). 

14. For convenience, Claim 15 of the above application is set out below with elements (g)-(n) 
identified as follows: 

An apparatus containing a set of processor readable instructions and communicating with a 
programmable device through at least one communication link, wherein the programmable 
device including at least one processor responsive to the set of processor readable instructions, 
wherein the set of instructions directs the programmable device to perform a method for creating 
multimedia meeting minutes, the method comprising the steps of: 

(g) receiving a plurality of notations from a notetaking user; 

(h) recording an index value for each of the plurality of notations, the index value based on the 
context of each notation; 

(j) receiving a quantity of multimedia information from at least one multimedia source; 

(k) selecting at least one portion of the quantity of the multimedia information based on the 
index value of each notation; 
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(1) creating an association between each of the plurality of notations and the selected portion 
of the quantity of multimedia information, where the association enables access to the 
selected portion of the quantity of multimedia information; 

(m) storing the plurality of notations and the respective associations for retrieval at a fixture 
time, where the future time is a time during the meeting; and 

(n) transmitting the plurality of notations and their respective associations via an electronic 
network to at least one user. 

15. The "Lite Minutes system" prototype embodiment of the invention contained all the 
elements of Claim 15 as described and claimed in the above U.S. patent application 

16. The "LiteMinutes system" is able to receive a plurality of notations from a notetaking user 
as claimed in Claim 15, element (g). 

17. The "LiteMinutes system" is able to record an index value for each of the plurality of 
notations, the index value based on the context of each notation as claimed in Claim 15, 
element (h). 

18. The "LiteMinutes system" is able to receive a quantity of multimedia information from at 
least one multimedia source as claimed in Claim 15, element (j). 

19. The "LiteMinutes system" is able to select at least one portion of the quantity of the 
multimedia information based on the index value of each notation as claimed in Claim 15, 
element (k). 

20. The "LiteMinutes system" is able to create an association between each of the plurality of 
notations and the selected portion of the quantity of multimedia information, where the 
association enables access to the selected portion of the quantity of multimedia information 
as claimed in Claim 15, element (1). 

21. The "LiteMinutes system" is able to store the plurality of notations and the respective 
associations for retrieval at a future time, where the future time is a time during the meeting 
as claimed in Claim 15, element (m). 

22. The "LiteMinutes system" is able to transmit the plurality of notations and their respective 
associations via an electronic network to at least one user as claimed in Claim 15, element 
(n). 

23. "LiteMinutes.java" and "DataStore.exe" as shown in Exhibits 'A'-'C were compiled prior 
to March 31, 2001. 
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24. "LiteMinutes.java" and "DataStore.exe" as shown in Exhibits 'A'-'C were completed 
versions of a prototype of the invention described and claimed in the above U.S. patent 
application 

25. "LiteMinutesjava" is a client program with associated executable ".CLASS" files listed on 
Exhibit 'A'. 

26. "DataStore.dsp" is a server program written in C++ listed on Exhibit 6 B\ 

27. "DataStore.exe" is the executable version of the server program listed on Exhibit *C\ 

28. A report on the "Lite Minutes: An Internet-Based System for Multimedia Meeting 
Minutes" prototype embodiment of the invention was prepared on March 9, 2000. 

29. The report on "Lite Minutes: An Internet-Based System for Multimedia Meeting Minutes" 
is shown in a redacted version as Exhibit 'D\ 

30. We signed the report on "Lite Minutes: An Internet-Based System for Multimedia Meeting 

Minutes" between the dates of March 9, 2000 and March 14, 2000. 

31. We recall receiving from Jonathan M. Hollander manuscripts and other correspondence 
concerning the prosecution of the above U.S. patent application 

32. We received a revised manuscript of the above U.S. patent application on March 27, 2001 

33. We responded with comments and/or corrections to the revised manuscript on April 10, 
2001. 

34. We received a request from Jonathan M. Hollander for our full names, addresses and 
citizenships as inventors of the above U.S. patent application 

35. We responded to the request for our full names, addresses and citizenships on April 11, 
2001. 

36. We received a request from Jonathan M. Hollander regarding our employment status and 
obligation to disclose and assign inventions regarding the above U.S. patent application 

37. We responded to Jonathan M. Hollander's request regarding our employment status and 
obligation to disclose and assign inventions on April 16, 2001. 

38. We received a prepared declaration and assignment from Jonathan M. Hollander for the 
above U.S. patent application 

39. We signed the declaration and assignment between the dates of April 17, 2001 and April 
23,2001. 
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40. The signed declaration and assignments were returned to Jonathan M. Hollander on April 
23,2001. 

41. This invention was first disclosed in a World Wide Web conference article first made 
publicly available on a non-confidential basis on May 1, 2001 

42. The undersigned, being hereby warned that willful false statements and the like so made 
are punishable by fine or imprisonment, or both, under 18 U.S.C. §1001, and that such 
willful false statements may jeopardize the validity of the application or any resulting 
registration, declares that the facts set forth in this declaration are true; all statements made 
of his/her own knowledge are true; and all statements made on information and belief are 
believed to be true. 
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Proposal Title: LiteMinutes: An Internet-Based System for Multimedia 
Meeting, Minutes 

Brief Description: Research in multimedia note taking technologies plus a survey that we 
conducted shows the need for an extremely lightweight system and support for both email and 
Web access to meeting minutes. LiteMinutes is a highly usable lightweight Internet-based 
multimedia meeting minutes system. It uses the Web and email for creating, revising, accessing 
and playing back meeting minutes. The note taking applet runs inside a Web browser on a laptop 
computer. A simple text box supports rapid interaction during a live meeting. At the end of the 
meeting, the notes are sent by the applet to the server, which formats them in HTML with links for 
video playback and slide images, and these meeting minutes are both emailed to the note taker 
and archived on the Web. The note taker revises the emailed meeting minutes in his email 
application and forwards them to all interested parties. Revisions are handled through email, and 
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EXHIBIT D 



the Web archive is automatically updated. The minutes can be accessed dynamically during a 
meeting, and slide images captured from earlier in the meeting may be shown on the main or 
secondary display in the meeting room. 

Description of Invention 

Introduction 

Documenting a meeting through meeting minutes often plays an important part in organizational activities. 
Meeting minutes constitute part of the organizational memory. Sometimes it is useful to look at meeting 
minutes right after a meeting to review and act on decisions. During a meeting, it can be helpful to show 
something from a point earlier in the meeting; for example, asking a question that refers to an earlier slide. 

New capabilities enabled by digital multimedia technology provide instant and wide communication of 
information. Video can pick up details that are difficult to catch, capture gestures, nonverbal activity, and 
show context Slides contain text, images and meaningful layout information. Meeting minutes, when 
correlated and linked to the video recording and slides, can be used to retrieve and playback interesting 
points of a meeting. By working on the Internet, meeting minutes can be created and accessed anywhere. 

Creating multimedia meeting minutes is a difficult challenge. Our extensive experience in multimedia note 
taking (see [3], [5], [1 1], [13]) indicates that a tool for taking meeting minutes must be lightweight and 
support rapid interaction. Taking notes in a live event requires the user to pay close attention and 
sometimes participate in the meeting in addition to formulating notes. This makes it very difficult for 
novice users of an application to fiddle with user interface widgets, modes, or perform tasks such as 
labeling or organizing information. 

In contrast, annotating video permits the design of a more complicated application. Video can be paused 
and replayed. This gives users time to operate the Ul and to perform labeling or organizing tasks. 
Consequently, unlike a meeting minutes system that must be used during a live event, a video annotation 
system can have many features and still be usable by non-experts. Examples of such systems are 
Microsoft MRAS [2], Marquee [12], Vanna [7]. 

Traditionally, meeting minutes are transcribed as text documents. People like text notes. Studies report 
that people prefer to read text over handwritten notes [6], and they find it easier to scan text than audio 
annotations [2J. From among the familiar as well as novel technologies for note-taking such as paper, 
CrossPad, palm computers, PDAs, pen-based notebook computers, and laptops, our experience over the 
past two years with note taking systems for meetings suggests that for novice users with minimal training, 
only paper or laptops are usable. A big part of the problem is that pen-based hardware still needs 
significant improvements before supplanting paper. We have done some exploration of paper technologies 
for multimedia note taking in earlier work [4]. The innovation that we are bringing to the laptop is a 
lightweight multimedia meeting minutes system that is Internet based, and this is the subject of this IP. 
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Empirical Observations and Design Requirements 

At our lab, text-based meeting minutes have been in use for the past two years. Notes were taken on paper 
by a single person, transcribed and sent out as an email to <all> in the lab. This was a tedious process, and 
the note taker often had to track down people in the meeting to clarify what was said or to obtain 
information that was shown on a slide. Part of this was due to the fact that it is difficult to catch 
everything in a meeting, About five months ago, welbegan using laptops for taking meeting minutes. This 
helped save time by not having to do transcription from paper notes. Since we also make video recordings 
and capture slide images from our meetings, and people have found the ability to playback points in a 
meeting and viewing slides to be useful, our goal was to design a meeting minutes application that would 
support linking to video and slides. 

We conducted a survey to see how people want to access the meeting minutes. We walked around the 
building and interviewed 13 people in their offices. They were asked two questions: 

(1) Do you read the email meeting minutes that you have been receiving? 

(2) Would you prefer to have the minutes: (a) emailed to you, (b) put on the Web, or (c) both? 
They were also given an opportunity to make comments following these questions. 

For question (1), 1 1 of the 13 subjects answered that they read the emails. One subject commented that he 
"looked more carefully if missed meeting." Another read them for the "spin." For question (2), 5 
preferred email, 2 preferred the Web, 5 preferred both, 1 said it "doesn't matter." One subject commented: 
"won't go to the Web to look, only email." Another commented that it was "easier to find things on the 
Web than through email," 

This survey suggests that it is important to have both email and Web access to the meeting minutes. 

The LiteMinutes System 

With the knowledge gained from aforementioned studies and our survey, we designed and built the 
LiteMinutes System, which has been in use for taking our staff meeting minutes for more than a month. In 
this section, we describe how the application is used to create and access multimedia meeting minutes. 
How the system works is explained in the Technical Details section below. 

Creating Meeting Minutes 

The LiteMinutes system is very easy to use: meeting participants or a designated scribe walks into a 
meeting with their laptops or uses ones supplied in the room. From a Web page, they click on the link for 
the note taking applet (see Figure 1), The laptops in the room, preferably connected to a wireless network, 
can be setup to run this applet permanently, so that in effect they are meeting room appliances. A Clear 
button allows the user wipe out notes left from earlier meetings (people do not always clean up after 
themselves, as evident by looking at whiteboards in meeting rooms). The user takes notes in a text 
window. There are no special keys or functions to learn. Meanwhile, the meeting is recorded digitally on 
video and the slide images are captured as they are shown by the presenters. 

At the end of the meeting, the user enters his email address and presses the Create Notes button. The 
notes are sent to the server, which is a CGI-Script, and parsed so that each note item separated by a 
newline character is identified with a timestamp corresponding to the time in the video recording when that 
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item was typed. These notes are formatted in HTML, with the small addition to the end of each item, 
where a [video] and/or [slides] link is created (this and several alternatives are shown in Figure 2). There 
is also a link at the end of the email to go to the Web archive and a link for Help. In appearance, they look 
very similar to the old email meeting minutes transcribed from paper of the past two^earstthese are 
identical to the plain-text version in Figure 2), so that there was minimal intrusion by the introduction of 
this system. 

These notes are emailed to the note taker (encoded in HTML and plain text) and a copy is archived on the 
Web for easy access. The note taker almost always revises the emailed meeting minutes. Revision is a 
crucial step in the meeting minutes process that allows details to be checked and typos to be fixed. For 
meetings that are more informal than staff meetings, revision may be skipped. This step is easily 
performed in the user's own email application, and the revision is then forwarded as an email to all the 
meeting participants and other interested parties. When a revision is detected by the system (explained in 
Technical Details section below), it updates the archive copy of the meeting minutes. 

An alternative way to do the revision is to go to the Web to edit the meeting minutes. We did prototyped 
this as both a form-based and an applet-based application. Our staff meeting scribe strongly prefers the 
email process. The emailed minutes serves as a reminder to revise them and she does not have to go look 
for them. Using email leverages an existing document routing system and meshes smoothly with existing 
work processes. 

Accessing the Meeting Minutes 

Multimedia meeting minutes can serve very different purposes depending on when they are accessed. 
Three important timeframes along with some examples that we have identified are: 

• During a meeting (real time) 

o Question & discussion referring to certain slides 
o Video access not supported (cf. Where Were We) 

• Right after a meeting (minutes to days) 

o Review details, action items 
o Skim email minutes 

• After an indefinite duration (months to years) 

o Organizational memory 

o Recall plans & accomplishments over past year 

The first one is the most complicated, and we will discuss that last. 

Right after a meeting, within minutes to days, the recipients of the email can skim the notes quickly, or 
playback the video by clicking on the video links at the end of interesting note items, which brings up a 
video player (see Figure 3). Clicking on a slide link shows the corresponding slide image, which is shown 
in the LiteSlide Viewer applet (see Figure 4). The archived notes, which are identical to the HTML- 
formatted email, may be accessed and browsed in a similar way. 

After an indefinite duration of months to years, the Web archive provides a better way to retrieve and 
access the meeting minutes. Currently we list sessions by the week, and this could be supplemented with 
search capabilities. People who faithfully file away their email minutes can also search through them. 
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During a meeting, any meeting participant can dynamically view the meeting minutes on the Web archive 
from their laptops as the scribe or others are taking notes. A person can dick on the slide link of a 
particular note item to see that slide on her laptop, which is displayed in our LiteSIide Viewer applet (see 
Figure 4). This applet also has two buttons for navigating back and forth among the slides. Furthermore, 
it has a button for the user to send this slide to a shared display in the meeting room, and a combo box for 
selecting the display. 

The ability of any participant to refer to and view an earlier slide during the meeting itself maps to a real 
and common task in meetings. An example is asking a question about a certain slide. A slide shown by 
any user who is not the presenter should be on a secondary display in a room with multiple displays, so as 
not to intrude upon the material on the main display being used by the presenter. In our conference room, 
we have a 120-inch rear projector as the main display and have acquired a smaller 42-inch plasma panel 
for the secondary display (see Figure 5). 

A simple applet, LiteShow (see Figure 5), runs inside a Web browser shown on the shared display and 
takes requests to show slide images from the LiteSIide Viewer applets. There needs to be restrictions on 
when such a request is valid; e.g. a person must be a participant in the meeting in the room or at distributed 
setting. A Clear button allows the user wipe out slide images left from earlier meetings. 

One may also consider playing back of the video recording during the meeting itself (see [8]), a capability 
that we felt is intriguing but not one that supports common tasks in a meeting. Our Metadata Media Player 
does not support this feature. 

Technical Details 

The LiteMinutes applet for taking meeting minutes is a Java 2 Swing applet. It runs in any Web browser 
that supports Java 2 or has a Java 2 plugin. The CGI-Script on the server side is written in C++, but it may 
be in other languages such as Perl, Python, or Java. When the user presses the Create Notes button, the 
text notes along with timestamps for each character, are sent from the applet to the server. The server 
parses and formats the notes in HTML with the video and/or slide links, emails a copy via a SMTP mail 
server, and archives a copy on the meeting minutes Web site. 

Revisions are handled when the user sends an email to <all> (which contains the email address 
<LiteMinutes>), or when <LiteMinutes> is included (e.g. cc:) along with the recipients of the minutes. 
The Web archive is automatically updated as follows. LiteMinutes has a mailbox, and minutes are 
compared to the Web archive files for similarity by looking at the message header, the time, and minor 
differences in message body. Messages that are deemed to be revisions are updated on the Web archive. 

The video is recorded directly in MPEG and can be played back right after a meeting. The MPEG player, 
called the Metadata Media Player, is built by FXPAL. It is possible to use any video player that provides 
an API function to play a video at a given point in time. One such popular format is RealVideo, which we 
also support. We note that an audio recording can be used instead of video. 

Slides are captured either by a screen capture component on the PC workstation whose monitor output is 
the meeting room large display, or alternatively by capturing images from the video monitor signal of the 
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laptops used for running the presentation application (details of this kind of technology are described in 
[5]). The system can support GIF and JPEG formats. The time that a particular slide is displayed by the 
presenter is recorded along with the slide image (e.g. encoded as part of the file name). The [slide] link of 
a note item is generated so that it points to the nearest slide preceding the time when that note item was 
created. With the LiteSlide Viewer applet, this link is an URL that accesses and displays the slide image 
file through the applet. 

The LiteShow applet is basically a window that displays the slide images (i.e. GIF or JPEG files), which 
accepts requests from the LiteSlide Viewer applets. One way to handle this is for this applet to check 
periodically (say every couple of seconds) with a central server that takes requests from the 
LiteSlide Viewer applets. 



Status 

LiteMinutes, with video links, has been implemented and deployed at FXPAL. It has been used for staff 
meetings with a high level of enthusiasm and satisfaction. LiteSlide Viewer and LiteShow are not yet 
functional. 
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Figure 1. Note taking applet client running inside a Web browser on a laptop. 
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Figure 2. Closeup of HTML formatted meeting minutes: (a) plain text, (b) with links to 
video, (c) with links to video and slides, (d) with thumbnail links. 
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Figure 3. Video playback from meeting minutes is activated by clicking on the rvideol 
links. 
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Figure 4. Slides images are accessed from the meeting minutes by clicking on the fslidel 
links. The slide images are shown in the LiteSlideViewer applet. 
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Figure 5. During the meeting itself, slide images can be displayed through the LiteShow 
applet on the main or secondary displays in the room. 
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